Cut-line steering methods for forming a mosaic image of a geographical area

ABSTRACT

Systems and methods are disclosed for creating a mosaic image of two or more geo-referenced source images, the geo-referenced source images having the same orientation, based on a ground confidence map created by analyzing pixels of one or more of the geo-referenced source images, the ground confidence map having values and data indicative of particular geographic locations represented by the values, at least one of the values indicative of a statistical probability that the particular geographic locations represented by the values represents the ground; and using routes for steering mosaic cut lines based at least in part on the values indicative of the statistical probability that the particular geographic locations represented by the values represents the ground of the ground confidence map, such that the routes have an increased statistical probability of cutting through pixels representative of the ground versus routes not based on the ground confidence map.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present patent application is a continuation of the patentapplication identified by U.S. Ser. No. 15/897,997, filed Feb. 15, 2018,which is a continuation of the patent application identified by U.S.Ser. No. 14/829,105, filed Aug. 18, 2015, which issued as U.S. Pat. No.9,898,802, which is a continuation of the patent application identifiedby U.S. Ser. No. 14/045,460, filed Oct. 3, 2013, which issued as U.S.Pat. No. 9,147,276, which is a continuation of U.S. Ser. No. 12/221,571,filed Aug. 5, 2008, which issued as U.S. Pat. No. 8,588,547, the entirecontents of each of which are hereby incorporated herein by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

In one version, the presently claimed and disclosed invention(s) relateto automated cut-line steering methods for forming an output mosaicimage of a geographical area. More particularly, in a preferredembodiment the presently claimed and disclosed invention(s) is anautomated cut-line steering method whereby separate, overlapping sourceimages are cut along preferred routes and then combined into at leastone single output mosaic image without requiring human intervention. Inaddition, a method is described and claimed which forms a groundconfidence map, useful in cut-line steering, formed by analyzingoverlapping portions of geo-referenced source images. The groundconfidence map provides an indication of where the overlapping portionsof the source images show ground locations. The ground confidence maphas a variety of uses such as determining preferred routes whereindividual source images are to be cut or correcting light detection andranging data (commonly known in the art as LIDAR). When used todetermine preferred routes, the ground confidence map maintains a highlevel of visual accuracy when the source images are combined to form atleast one single output mosaic image. The at least one single outputmosaic image is visually pleasing and geographically accurate.

2. Background of the Art

In the remote sensing/aerial imaging industry, imagery is used tocapture views of a geographic area and to be able to measure objects andstructures within the images as well as to be able to determinegeographic locations of points within the image. These are generallyreferred to as “geo-referenced images” and come in two basic categories:

-   -   1. Captured Imagery—these images have the appearance they were        captured by the camera or sensor employed.    -   2. Projected Imagery—these images have been processed and        converted such that they conform to a mathematical projection.

All imagery starts as captured imagery, but as most software cannotgeo-reference captured imagery, that imagery is then reprocessed tocreate the projected imagery. The most common form of projected imageryis the ortho-rectified image. This process aligns the image to anorthogonal or rectilinear grid (composed of rectangles). The input imageused to create an ortho-rectified image is a nadir image—that is, animage captured with the camera pointing straight down.

It is often quite desirable to combine multiple images into a largercomposite image such that the image covers a larger geographic area onthe ground. The most common form of this composite image is the“ortho-mosaic image” which is an image created from a series ofoverlapping or adjacent nadir images that are mathematically combinedinto a single ortho-rectified image.

Each input nadir image, as well as the output ortho-mosaic image, iscomposed of discrete pixels (individual picture elements) of informationor data. As part of the process for creating an ortho-rectified image,and hence an ortho-mosaic image, an attempt is made to reproject (movewithin a mathematical model) each pixel within the image such that theresulting image appears as if every pixel in the image were a nadirpixel —that is, that the camera is directly above each pixel in theimage.

The reason this ortho-rectification process is needed is it is notcurrently possible to capture an image where every pixel is nadir to(directly below) the camera unless: (1) the camera used is as large asthe area of capture, or (2) the camera is placed at an infinite distanceabove the area of capture such that the angle from the camera to thepixel is so close to straight down that it can be considered nadir. Theortho-rectification process creates an image that approximates theappearance of being captured with a camera where the area on the groundeach pixel captures is considered nadir to that pixel, i.e. directlybelow that pixel. This process is done by creating a mathematical modelof the ground, generally in a rectilinear grid (a grid formed ofrectangles), and reprojecting from the individual captured camera imageinto this rectilinear grid. This process moves the pixels from theirrelative non-nadir location within the individual images to their nadirpositions within the rectilinear grid, i.e. the image is warped to lineup with the grid.

When creating an ortho-mosaic, this same ortho-rectification process isused, however, instead of using only a single input nadir image, acollection of overlapping or adjacent nadir images are used and they arecombined to form a single composite ortho-rectified image known as anortho-mosaic. In general, the ortho-mosaic process entails the followingsteps:

-   -   A rectilinear grid is created, which results in an ortho-mosaic        image where every grid pixel covers the same amount of area on        the ground.    -   The location of each grid pixel is determined from the        mathematical definition of the grid. Generally, this means the        grid is given an X and Y starting or origin location and an X        and Y size for the grid pixels. Thus, the location of any pixel        is simply the origin location plus the number of pixels times        the size of each pixel. In mathematical terms:        X_(pixel)=X_(origin)+X_(size)×Column_(pixel) and        Y_(pixel)=Y_(origin)+Y_(size)×ROW_(pixel).    -   The available nadir images are checked to see if they cover the        same point on the ground as the grid pixel being filled. If so,        a mathematical formula is used to determine where that point on        the ground projects up onto the camera's pixel image map and        that resulting pixel value is then transferred to the grid        pixel. During this selection process, two important steps are        taken:    -   When selecting the image to use to provide the pixel value, a        mathematical formula is used to select an image that minimizes        building lean—the effect where buildings appear to lean away        from the camera. This is accomplished in a number of ways, but        the most common is to pick the image where the grid pixel        reprojects as close to the camera center, and hence as close to        that camera's nadir point, as possible.    -   When determining the source pixel value to use, the ground        elevation is taken into account to ensure the correct pixel        value is selected. Changes in elevation cause the apparent        location of the pixel to shift when captured by the camera. A        point on the ground that is higher up will appear farther from        the center of the image than a point on the ground in the same        location that is lower down. For instance, the top of a building        will appear farther from the center of an image than the bottom        of a building. By taking the ground elevation into account when        determining the source pixel value, the net effect is to        “flatten” the image out such that changes in pixel location due        to ground elevation are removed.

Because the rectilinear grids used for the ortho-mosaic are generallythe same grids used for creating maps, the ortho-mosaic images bear astriking similarity to maps and as such, are generally very easy to usefrom a direction and orientation standpoint. However, since they have anappearance dictated by mathematical projections instead of the normalappearance that a single camera captures and because they are capturedlooking straight down, this creates a view of the world to which we arenot accustomed. As a result, many people have difficulty determiningwhat it is they are looking at in the image. For instance, they mightsee a yellow rectangle in the image and not realize what they arelooking at is the top of a school bus. Or they might have difficultydistinguishing between two commercial properties since the only thingthey can see of the properties in the ortho-mosaic is their roof tops,where as most of the distinguishing properties are on the sides of thebuildings. An entire profession, the photo interpreter, has arisen toaddress these difficulties as these individuals have years of trainingand experience specifically in interpreting what they are seeing innadir or ortho-mosaic imagery.

Since an oblique image, by definition, is captured at an angle, itpresents a more natural appearance because it shows the sides of objectsand structures—what we are most accustomed to seeing. In addition,because oblique images are not generally ortho-rectified, they are stillin the natural appearance that the camera captures as opposed to themathematical construction of the ortho-mosaic image. This combinationmakes it very easy for people to look at something in an oblique imageand realize what that object is. Photo interpretation skills are notrequired when working with oblique images.

Oblique images, however, present another issue. Because people havelearned navigation skills on maps, the fact that oblique images are notaligned to a map grid, like ortho-mosaic images, makes them much lessintuitive when attempting to navigate or determine direction on animage. When an ortho-mosaic is created, because it is created to arectilinear grid that is generally a map grid, the top of theortho-mosaic image is north, the right side is east, the bottom issouth, and the left side is west. This is how people are generallyaccustomed to orienting and navigating on a map. But an oblique imagecan be captured from any direction and the top of the image is generally“up and back,” meaning that vertical structures point towards the top ofthe image, but that the top of the image is also closer to the horizon.However, because the image can be captured from any direction, thehorizon can be in any direction, north, south, east, west, or any pointin between. If the image is captured such that the camera is pointingnorth, then the right side of the image is east and the left side of theimage is west. However, if the image is captured such that the camera ispointing south, then the right side of the image is west and the leftside of the image is east. This can cause confusion for someone tryingto navigate within the image.

Additionally, because the ortho-mosaic grid is generally a rectilineargrid, by mathematical definition, the four cardinal compass directionsmeet at right angles (90-degrees). But with an oblique image, because itis still in the original form the camera captured and has not beenre-projected into a mathematical model, it is not necessarily true thatthe compass directions meet at right angles within the image. Because inthe oblique perspective, you are moving towards the horizon as you moveup in the image, the image covers a wider area on the ground near thetop of the image as compared to the area on the ground covered near thebottom of the image. If you were to paint a rectangular grid on theground and capture it with an oblique image, the lines along thedirection the camera is pointing would appear to converge in thedistance and the lines across the direction of the camera is pointingwould appear to be more widely spaced in the front of the image thanthey do in the back of the image. This is the perspective view we areall used to seeing—things are smaller in the distance than close up andparallel lines, such as railroad tracks, appear to converge in thedistance. By contrast, if an ortho-mosaic image was created over thissame painted rectangular grid, it would appear as a rectangular grid inthe ortho-mosaic image since all perspective is removed as an incidentalpart of the ortho-mosaic process.

Because of these fundamental differences in perspective and appearance,the creation of an ortho-mosaic image by the process described abovedoes not work well for oblique images. Because the camera's optical axis(an imaginary line through the center of the lens or optics that followsthe aim of the camera) is typically pointed at an angle of 45-degrees ormore from nadir (pointed 45-degrees or more up from straight down), theeffects of building lean, elevation differences, and non-square pixelsare all exaggerated—effects that are considered negative qualities in anortho-mosaic image. In the ortho-mosaic industry, requirements aregenerally placed on the image capture process such that they limit theamount of obliqueness to as little as 5-degrees from nadir so as tominimize each of these negative effects.

In addition, if the admirable properties of an oblique image are to bemaintained, namely seeing the sides of structures and the naturalappearance of the images, then clearly a process that attempts to removevertical displacements, and hence the sides of the buildings, and onethat warps the image to fit a rectilinear grid is not a viable choice.In order to maintain the admirable qualities of the oblique image, itmay be necessary that the process:

-   -   If the oblique perspective is to be maintained, the pixels        cannot be aligned to a rectilinear grid, or even a trapezoidal        grid. Instead, the pixels are preferably aligned to the natural        perspective that a camera captures.    -   As part of the oblique perspective, the pixels in the image        cannot all measure the same size on the ground, as pixels in the        foreground of the image cover a much smaller area on the ground        than pixels in the background of the image—that is by definition        part of the natural perspective of a camera.    -   Because the pixels are so far from nadir, the effects of        building lean become extreme and the standard solutions employed        in the ortho-mosaic process do not do an adequate enough job        compensating for this effect—new techniques must be developed to        better compensate for this effect.    -   If the effects of changes in elevation are backed out, the        resulting image has a very unnatural appearance—the vertical        sides of buildings can warp and twist, which is something we are        not accustomed to seeing and therefore, when looking at such an        image, we have a tendency to “reject” it. Thus, to keep the        buildings, structures, and objects within an image looking        natural, it is preferable to leave the effects of elevation in        the perspective of the image and instead account for it in        another manner.

Because of these issues, the common practice in the industry is toprovide oblique imagery as a series of individual images. However, someof the same benefits of the ortho-mosaic also apply to an oblique-mosaic(an image created from a collection of overlapping or adjacent obliqueimages), namely the fact that the mosaic covers a larger geographic areathan each or any of the individual images that were used to create it.

SUMMARY OF THE INVENTION

This invention allows for the creation of an output mosaic image thathas both a natural appearance and is preferably geo-referenced tomaintain the ability to measure and determine geographic coordinates.While the preferred embodiment applies this invention to aerial obliqueimagery, the invention will also work with non-aerial oblique imagerycaptured in a variety of ways, including but not limited to camerasmounted obliquely on a vertical pole, hand-held cameras aimed obliquely,and cameras mounted at oblique angles on an underwater probe. While thepreferred embodiment is used for cut-line steering when creating obliquemosaics, this invention will also work for cut-line steering forortho-mosaics as well using input nadir images. This method, especiallywhen utilizing the ground confidence map, can also be applied to “streetside” imagery (images captured horizontally—typically from a movingvehicle or by pedestrians), with the slight modification of using thebuilding fronts in a similar fashion as the ground is used when thisinvention is used in conjunction with aerial imagery.

In one embodiment, the present invention is a method for automaticallysteering mosaic cut lines along preferred routes to form an outputmosaic image. An area to be represented by the output mosaic image isselected, and then an assignment map having a plurality of pixelassignments corresponding to the output mosaic image is created. Thepixel assignments have an initial designation of unassigned. Then, eachpixel assignment of the assignment map that intersects the preferredroutes is designated as a Preferred Cut Line pixel, which has the effectof dividing the Assignment Map into one or more regions that are boundedby Preferred Cut Line pixels or the edge of the Assignment Map. For eachregion, one or more source images that completely cover the region areselected, and for each selected source image, a Selection Heuristic isused to determine the quality of coverage, and then each pixelassignment in that region is designated as being assigned to the imagewith the best heuristic.

For any remaining unassigned regions, two or more source images areselected whose combined area completely covers the region, and for eachset of two or more combined images, a Pairing Heuristic is used todetermine the quality of coverage. Then each pixel in the region isdesignated as being assigned to the two or more combined images with thebest heuristic.

The Preferred Cut Line pixels are re-designated to match the imageassignments of their bounded regions, and then pixel values from thesource images corresponding to the pixel assignments are utilized tocreate the output mosaic image. In one embodiment, this can beaccomplished by stepping through each pixel in the Assignment Map andusing the stored image assignment to determine which image or images touse for the actual image content of the output mosaic image.

In another aspect, the presently disclosed and claimed invention isdirected to a method of cut-line steering by creating a groundconfidence map (shown in FIG. 5) of a geographic area. The groundconfidence map shows which areas of the overlapping sources images arerepresentative of a ground location and which are not, which minimizesthe likelihood that the preferred routes will be steered through athree-dimensional object when forming the output mosaic image.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is an exemplary color output mosaic image constructed inaccordance with one embodiment of the present invention and formed fromeleven separate source images.

FIG. 2A is a flow chart illustrating an exemplary method for creatingthe output mosaic image in accordance with the present invention.

FIG. 2B is a continuation of the flow chart depicted in FIG. 2A.

FIG. 3 is an illustration of an exemplary assignment map for the outputmosaic image that has been marked with preferred routes for cutting thesource images which has the effect of dividing the assignment map intoone or more regions that are bounded by the preferred routes or the edgeof the assignment map.

FIG. 4 is a diagrammatic view of the assignment map of FIG. 3 showingcoverage of certain regions by three source images.

FIG. 5 is an exemplary color ground confidence map constructed inaccordance with one embodiment of the present invention.

FIG. 6 is a schematic representation of capturing geo-referenced, colordigital source images highlighting a plurality of kernels located on theground of a geographic area.

FIG. 7 depicts exemplary color source images captured from differentvantage points showing the plurality of kernels depicted in FIG. 6.

FIG. 8 is an exemplary color pixel image of one of the kernels capturedfrom the different vantage points in FIG. 6.

FIG. 9 is an exemplary color pixel image of another kernel captured fromthe different vantage points in FIG. 6.

DETAILED DESCRIPTION OF THE PRESENTLY DISCLOSED AND CLAIMED INVENTION

Before explaining at least one embodiment of the invention in detail, itis to be understood that the invention is not limited in its applicationto the details of construction, experiments, exemplary data, and/or thearrangement of the components set forth in the following description orillustrated in the drawings. The invention is capable of otherembodiments or of being practiced or carried out in various ways. Also,it is to be understood that the phraseology and terminology employedherein is for purpose of description and should not be regarded aslimiting.

The presently claimed and disclosed invention(s) relate to mosaic imagesand methods for making and using the same. More particularly, thepresently claimed and disclosed invention(s) use a methodology forautomatically steering mosaic cut lines along preferred routes to forman output mosaic image whereby separately captured images (referred tohereinafter as “source images”) are automatically combined into at leastone single mosaic image. The at least one single mosaic image isvisually pleasing and geographically accurate. The source images arepreferably aerial images and can be either nadir images, orthogonalimages, or oblique images.

Referring now to the Figures and in particular to FIG. 1, shown thereinand designated by a reference numeral 10 is an exemplary output mosaicimage constructed in accordance with one embodiment of the presentinvention and formed from contributing pixels of twelve separatelycaptured geo-referenced source images, designated by referencenumerals_16 a-16 l. While FIG. 1 depicts the source images 16 as beingprimarily nadir (vertical) in their orientation, it should be understoodthat the source images 16 can be oriented in a variety of differentways, including, but not limited to, oblique and horizontalorientations. Preferably every pixel of the geo-referenced source images16 is associated with a geographic location of a point within the image.The source images 16 can be geo-referenced utilizing any suitabletechnology. For example, one method of geo-referencing the source imagesis described in U.S. Ser. No. 10/701,839, filed on Nov. 5, 2003 andentitled “METHOD AND APPARATUS FOR CAPTURING GEOLOCATING AND MEASURINGOBLIQUE IMAGES”. The white lines on the output mosaic image 10illustrate the transitions from contributing pixels of the source images16.

Shown on the output mosaic image 10 are “preferred routes” 24 for“cutting” the source images 16 to form the output mosaic image 10. Thepreferred routes 24 are selected so as to minimize any adverse effectswhen transitioning between adjacent source images 16. Preferably, thepreferred routes 24 are selected in areas where there are no structuresabove or below the ground elevation model. This can be accomplished byplacing the preferred route 24 down the middle of a street, or by usinga ground confidence map as described below. In the exemplary FIG. 1, thepreferred routes 24 are generated from street centerline informationbecause streets are generally close to ground level, and do not normallyrun through vertical structures such as buildings or trees. Thus, ifthere is a street in the area where two source images 16 overlap, thenthe transition from contributing pixels from one source image 16 tocontributing pixels from an adjacent source image 16 may occur alongthis street, thus minimizing the adverse effects associated withtransitioning between the contributing pixels of the source images 16 inthe output mosaic image 10. The street centerlines can be obtained, forexample, from vector data files such as TIGER files or other GeographicInformation System files. It should be understood that the preferredroutes 24 can be generated from other sources besides streetcenterlines.

The preferred routes 24 and transition lines 28 are shown on the outputmosaic image 10 of FIG. 1 for purposes of showing how the output mosaicimage 10 was constructed. It should be understood that the preferredroutes 24 and the transition lines 28 will not usually be shown in theoutput mosaic image 10 constructed in accordance with the presentinvention.

FIGS. 2A and 2B depict a logic flow chart 30 of an automated cut linesteering algorithm constructed in accordance with the present inventionand stored on a computer readable medium. The automated cut linesteering algorithm is adapted to execute on a computer system or systemsand create the output mosaic image 10 preferably without any manualintervention.

FIGS. 3 and 4 cooperate to show certain steps in the formation of theoutput mosaic image 10, and provide visual representations of the logicflow provided in FIGS. 2A and 2B. In general, as indicated by a block32, a desired area is selected to be represented by one or more outputmosaic image(s) 10. The desired area is preferably manually selected,although automated selection of the desired area is also contemplated.Once the desired area is selected, a number of output mosaic images 10to represent the desired area or a size of each output mosaic image 10can be selected. For example, the desired area could be Los AngelesCounty, and the size of each output mosaic image 10 could be specifiedas one square mile. In this example, the automated cut line steeringalgorithm would proceed to create an output mosaic image 10 for eachsquare mile of Los Angeles County. Generally, the area to be representedby one or more output mosaic image(s) 10 would be a specificgeographical location. However, other areas can also be selected to beimaged into the output mosaic image 10 such as building sides, walls,landscapes, mountain sides and the like.

As shown in the logic flow chart 30 of FIGS. 2A and 2B, once the desiredarea is selected, the source images 16 are obtained as indicated byblock 36. However, it should be understood that the source images 16 canbe obtained prior to selection of the desired area, stored on one ormore computer readable medium and then accessed. In general, the sourceimages 16 are preferably obtained utilizing one or more real camerascapturing the source images 16 of portions of the desired area and thengeo-referenced as discussed above and optionally color-balanced.

The output mosaic image 10 is initially formed by creating an assignmentmap 41 corresponding to the output mosaic image 10, as indicated byblock 40 of FIG. 2A. An exemplary assignment map 41 is shown in FIG. 3.The assignment map 41 is provided with output edges 42 surrounding anassignment area 43 with a plurality of pixels, denoted by the dashedlines and arranged in a rectilinear grid format covering the assignmentarea 43.

Initially, every pixel of the assignment map 41 preferably has aninitial designation of unassigned. Then, as shown in block 44 of thelogic flow chart 30, pixels, such as each pixel, of the assignment map41 that intersect a preferred route 24 is marked as being a “preferredcut line pixel”, which has the effect of dividing the assignment map 41into one or more regions 45 that are bounded by preferred cut linepixels 46 or the output edges 42 of the assignment map 41. By way ofexample, six regions 45 are depicted in FIG. 3 and labeled with thereference numerals 45 a-f. As indicated by block 48, the remaining stepsof the logic flow chart 30 are to be performed on each region 45. Thepreferred cut line pixels 46 cover cut line areas represented by thepreferred routes 24 and such cut line areas have a length and a width.It should be noted that the width of the preferred routes 24 can bevaried depending upon design factors, such as the amount of featheringto be accomplished between the transitions from adjacent source images.

A region 45 is a contiguous set of pixels bounded by preferred cut linepixels 46 or the output edges 42. In the exemplary assignment map 41depicted in FIG. 3, the assignment map 41 is divided into six regionsthat are designated with the reference numerals 45 a-45 f. The markingof the preferred routes 24 can be accomplished in any suitable manner,such as by drawing a vector representation of the street centerlinesonto the assignment map 41 thereby converting the vector representationof the street center lines into a raster representation. Or, thesepreferred cut line pixels 46 can be generated from raster form of data,such as by using those pixels in a ground confidence image whose groundconfidence value meets or exceeds a particular threshold.

It is generally desirable to create a continuous output mosaic image 10.In order to do so, there must be source images 16 for the entireassignment area 43 being depicted in the output mosaic image 10. Morespecifically, in order to create a preferred embodiment of the mosaicoutput image 10, all of the regions 45 preferably are assigned at leastone source image as indicated in block 52. This means that if multiplesource images 16 are being combined to create the output mosaic image10, the source images 16 must be adjacent or more commonly, overlapping.FIG. 4 shows the exemplary embodiment of the assignment map 41 of FIG. 3with two overlapping source images (designated with reference numerals16 l-16 m) assigned to a portion of the regions 45. While theoverlapping source images 16 l-16 m are depicted in FIG. 4 as beingoverlapping nadir source images 16, it should be understood that thesource images 16 can be in a variety of orientations, including, but notlimited to, oblique and orthogonal and/or nadir orientations. Inaddition, the use of two source images 16 in FIG. 4 is only toaccentuate the overlapping assignment of source images 16 to theassignment map 41. As previously described in relation to FIG. 1, thenumber of sources images 16 used in the assignment can be of any number.

As a result of this overlap, it is common for there to be multiplesource images 16 covering the same area on the ground. If multiplecaptured source images 16 are available for selection, a preferredcaptured source image 16 is chosen according to the selection criteriadescribed below. In general, the method attempts to minimize the numberof source images 16 assigned to a given region 45 in order to minimizethe number of cut-lines within the particular region 45. Thus, for eachregion 45 (as noted above with reference to block 48 of the logic flow),one or more source images 16 are preferably located in an orientationthat allows the source images 16 to completely cover the region 45, asindicated by branching decision block 56. When the source images 16 areaerial images, the ground location for the boundaries of the region 45is determined, which can then be used to ascertain and select whichsource images 16 contain image data for that particular identifiedground location. This is generally done by checking to see if the groundlocation lies within the image boundaries of a previously capturedsource image 16. In the example shown in FIG. 4, source image 16 lcompletely covers the region 45 b, and the source image 16 m completelycovers the regions 45 e and 45 f. If a source image 16 is not locatedthat completely covers the one or more region 45, then the automated cutline steering algorithm determines whether two or more of the sourceimages 16 combine to completely cover the particular region 45, asindicated by branching decision block 60.

As indicated by branching decision block 64 and block 68, if more thanone source image 16 is located for a particular region 45, then aSelection Heuristic for quality of coverage can optionally be utilizedto determine which source image 16 to select for contributing pixels tothe region 45. A variety of Selection Heuristics can be utilized and thefollowing are discussed below by way of example. The SelectionHeuristics can be selected from the group comprising (1) which sourceimage 16 is closest to nadir in the area covering the region 45, (2) thefirst source image 16 located within a region 45, (3) a source image 16that covers the largest number of surrounding preferred cut line pixels46, and (4) a source image 16 that covers the largest number of otherregions 45. As shown in box 72, once a Selection Heuristic selects aparticular source image 16 for the region 45, pixels in the region 45are designated as being assigned to particular pixels or groups ofpixels of the selected source image 16.

As shown in block 70, if it is determined that a source image 16completely covers a region 45 (branched decision block 56), then thesingle source image 16 is selected and the pixels in the region 45 aredesignated as being assigned to particular pixels or groups of pixels ofthe selected source image 16

As discussed above and shown in branching decision block 60, if a sourceimage 16 is not located that completely covers the region 45 such asregion 45 d shown in FIG. 4, then the automated cut line steeringalgorithm determines whether two or more of the source images 16 combineto completely cover the particular region 45 which in the example shownare source images 16 l-m. If two or more source images 16 combine tocover the region 45, then the automated cut line steering algorithm usesa Pairing Heuristic to attempt to enhance the quality of coverage of theregion 45 as shown in block 76. A variety of Pairing Heuristics can beutilized and the following are discussed below by way of example. Thefollowing are examples of Pairing Heuristics that can be utilized.

1. Find the source image 16 that covers the largest number of pixels inthe region 45; then, find the source image 16 that covers the largestnumber of remaining uncovered pixels in region 45; Continue until allpixels are covered.

2. Find source image 16 that is closest to nadir at center point ofregion 45. Mark all pixels in region 45 covered by this source image 16as “covered.” For each uncovered sub-region, find the source image 16that is closest to nadir and mark the region 45 pixels covered by thatsource image 16 as “covered.” Preferably, this method is repeated untilall pixels have been designated as “covered.”

3. Review list of possible source images 16 in order found, markingcoverage until all pixels are covered.

4. Expand preferred cut lines until sub-regions are created that aresmall enough to be covered by a single source image 16. Use a singlesource image 16 assignment method to select the source image 16 for eachof the new regions 45.

5. When selecting source images 16, the relative orientation of thecamera that captured the source images 16 can be taken into account,e.g., in order to achieve a more desirable output mosaic image 10,source images 16 that were captured in the same, or nearly the same,relative orientation of the virtual camera, in terms of oblique downwardangle and compass direction of the optical axis will be more compatible.

6. The type of camera can also be taken into account when selectingsource images 16. That is, if the type of camera utilized the capturethe source images 16 is radically different (for instance, a linescanner versus a full frame capture device), it may result in anundesirable resulting output mosaic image 10.

Once the Pairing Heuristic determines the coverage of the regionutilizing multiple source images 10, the automated cut-line steeringalgorithm then designates particular parts of the source images 16 tothe pixels in the assignment map 41, as indicated in block 80. This canbe accomplished in a variety of manners and the following are examplesof how this can be implemented.

1. As the source images 16 are selected, assign all unassigned pixels inthe region 45 that are covered by the current source image 16 to thatsource image 16.

2. Assign pixels from the source images 16 based on each pixel'snearness to nadir.

3. Assign pixels from the source images 16 based on the number ofsurrounding preferred cut line pixels 46 covered by the source image 16.

As shown in block 82, if two or more source images 16 do not combine tocompletely cover a region 45 of the assignment map 41, then additionalsources images may be obtained for each region that is designated asunassigned or the boundaries of the assignment map 41 can be adjusted.

As shown in block 84, once the pixels of the regions 45 have beendesignated or assigned to particular pixels or groups of pixels of thesource images 16, the preferred cut line pixels 46 are thenre-designated to match the source image 16 assignments of their boundedregions 45. As will be discussed below, this can be accomplished in avariety of manners and the method utilized for such re-designation maybe dependent on whether or not a single source image 16 covers adjacentregions 45 separated by the preferred cut line pixels 46. The followingare examples of how this can be accomplished.

1. If cut line area is only one pixel thick (as shown by way of examplein FIGS. 3 and 4), assignment of cut line area could be combined withassignment of adjacent region 45.

2. Reduce cut line area to one pixel thick, and then combine withassignment of adjacent region 45.

3. Work outward from assigned regions 45 into preferred cut line area,assigning region's 45 source image 16 to preferred cut line pixel if itis covered by the source image 16.

4. For each preferred cut line pixel 46, assign pixels from source image16 that is nearest to nadir, choosing from source images 16 that areassigned to one of the adjacent regions 45 if possible.

As indicated by block 96, once all of the pixels in the assignment map41 are designated or assigned particular pixels or groups of pixels fromthe source images 16, the output mosaic image 10 is created bycontributing the designated or assigned pixel values to the outputmosaic image 10, as indicated by block 88. This can be accomplished in avariety of manners and the following are merely examples of how this canbe accomplished.

1. This could be done either with or without feathering (shown in block92). Feathering makes the sharp changes occurring at the cut line appearmore gradual by altering pixel values at the cut line with a blend ofthe pixel values from each of the source images. For example, if afeathering region along a cut-line between source image 16 l, forexample, and source image 16 m is 4-pixels wide, then the first pixel inthe feathering region might by comprised of 20% of the value from thecorresponding pixel in source image 16 l and 80% of the value from thecorresponding pixel in source image 16 m, the second pixel in thefeathering region might by comprised of 40% of the value from thecorresponding pixel in source image 16 l and 60% of the value from thecorresponding pixel in source image 16 m, the third pixel in thefeathering region might by comprised of 60% of the value from thecorresponding pixel in source image 16 l and 40% of the value from thecorresponding pixel in source image 16 m, and the fourth pixel in thefeathering region might by comprised of 80% of the value from thecorresponding pixel in source image 16 l and 20% of the value from thecorresponding pixel in source image 16 m.

2. The contribution of a source image 16 to a given mosaic pixel couldbe determined using a nearest neighbor method, or based on averaging orinterpolating source image 16 pixel values.

A surface location is preferably assigned to each pixel included in theassignment map 41 so that the output mosaic image 10 will begeo-referenced.

In practice, the methodology disclosed and claimed herein, consists ofmultiple steps and data transformations that can be accomplished by oneof ordinary skill in the art given the present specification. Inaddition, follow-on work could create new algorithms specificallydesigned to deal with the complexities of source images 16, including,but not limited to, orthogonal, oblique, and/or nadir source images 16.

The current invention also contemplates a method of cut-line steering bycreating a ground confidence map 100 (shown in FIG. 5) of a geographicarea. The ground confidence map 100 shows which areas of the overlappingsources images 16 are representative of a ground location and which arenot. In the exemplary map 100, the red areas 101 are indicative ofground locations while the black pixels 102 are not. In general, the cutline steering method utilizing the ground confidence map 100 increasesthe statistical probability that preferred routes 24 used to transitionbetween various source images 16 to form the output mosaic image 10 arelocated on the ground rather than on or through a three-dimensionalobject, such as, but not limited to, an automobile, building, or tree.This is primarily accomplished by using at least one kernel 104 (shownas 104 a and 104 b) to compare the pixel values of overlapping sourceimages 16 to establish the statistical probability that the geographicallocation represented by pixels in the source images 16 actuallyrepresents the ground. While the kernels 104 are shown as being circularin shape in FIGS. 5-9, it should be understood that the kernels can beof any fanciful shape, including, but not limited to square,rectangular, ovular, or triangular.

Referring now to FIG. 6, shown therein is a schematic diagram depictingthe capturing of geo-referenced source images 16 from a plurality ofdifferent vantage points. While FIG. 6 shows the use of two cameras 105a and 105 b for capturing the geo-referenced source images 16 fromdifferent vantage points, it should be understood that the capturing canbe done from one or more camera(s) 108 as long as the camera(s) 108capture(s) the source images 16 from a variety of different vantagepoints. The cameras 105 a and 105 b utilize different vantage points 106a, 106 b, 107 a, and 107 b to capture source images 16 that areassociated with common kernels 104 a and 104 b. While the Figuresindicate the identification of two kernels 104 a and 104 b, it should beunderstood that the invention contemplates the identification of atleast one kernel 104 and is not limited to a specific number of kernels104 identified within the source images 16.

When trying to combine source images 16, a problem arises when theoverlapping portions of source images 16 captured from different vantagepoints 106 a, 106 b, 107 a, and 107 b represent structures that are noton the actual ground. This problem is illustrated by the camera 105 a inwhich the vantage points of the camera 105 a for a plurality of pixelsis blocked or shaded by a building 112. The effect of this shading isthe capturing of a source image 16 in which the building is shown, butupon geo-referencing the source images 16, the pixels representing theground location actually show the roof of the building 112 as shown inFIG. 7. That is, FIG. 7 shows two source images 16 which have beencaptured from two different vantage points. In the source image 16 onthe left, the kernel 104 a is represented on the roof of the building112, while in the source image 16 on the right, the kernel 104 a isrepresented on the ground and the pixels are very different inappearance. In contrast, the kernel 104 b in both source images isrepresented on the ground and the pixels within such kernel 104 b aresimilar in appearance. This has deleterious effects when establishingpreferred routes 24 for combining source images 16 into an output mosaicimage 10 as it is undesirable for the preferred routes 24 to run throughthe building 112 (or another three-dimensional structure).

In one aspect, the present invention is directed to solving this problemby creating the ground confidence map 100 in which the ground confidenceis shown in FIG. 5. After the ground confidence map 100 of a geographicregion is created, each pixel of the ground confidence map 100 can beassigned with a pixel value indicative of a ground confidence score bydetermining whether various points of overlapping portions of the sourceimages 16 represent the same physical object. This can be accomplishedby calculating a ground confidence score for pixel values of pixelslocated within the kernel 104 within the overlapping source images 16corresponding to a particular geographic location of the pixel withinthe source images 16. The kernel 104 is a small matrix of pixels,usually no larger than 9×9, that is used as an operator duringcomparison of the overlapping source images 16. The ground confidencescore is indicated by analyzing the pixel score for each pixel locatedwithin kernel 104 to develop a composite pixel score. For example, thepixel scores can be summed or averaged to develop the composite pixelscore for the kernel 104. The pixel score associated with a particularpixel located within the kernel 104 can be calculated in accordance withthe following formula:

P _(s) =|P _(a) −P _(b)|/(P _(a) +P _(b))   (1)

where P_(s) is the pixel score associated with a pixel located withinthe kernel 104, P_(a) is the pixel value of a pixel located within thekernel 104 captured by camera 105 a (Camera A) and indicative of aparticular color, and P_(b) is the pixel value of the same pixel locatedwithin the same kernel 104 captured by camera 105 b (Camera B) andindicative of a particular color. So, for instance, if a 3×3 kernel isbeing utilized, the formula would be:

$\begin{matrix}{{{Ps}\lbrack {r,c} \rbrack} = {\sum\limits_{i = {\pm 1}}{\sum\limits_{j = {\pm 1}}{{{{P\; {a\lbrack {{r + i^{\prime}},{c^{\prime} + j}} \rbrack}} - {{Pb}\lbrack {{r^{''} + i^{\prime}},{c^{''} + j}} \rbrack}}} \div ( {{P\; {a\lbrack {{r^{\prime} + i},{c^{\prime} + j}} \rbrack}} + {{Pb}\lbrack {{r^{''} + i},{c^{''} + j}} \rbrack}} )}}}} & (2)\end{matrix}$

-   Where Σ denotes summation,-   r=row number, c=column number,-   Ps[r,c] indicates the pixel score at that particular row and column-   Pa[r′,c′] indicates the input image A pixel that corresponds to the    location of Ps[r,c],-   Pb[r″,c″] indicates the input image B pixel that corresponds to the    location of Ps[r,c],-   Note that r !=r′ !=r″ and c !=c′ !=c″ (!=means not equals) but that    all three locations [r,c], [r′,c′] and [r″,c″] map to the same    location in the assignment map.-   Pa[r′+i,e+j] indicates the input image A pixel that is offset from    the corresponding pixel Pa[r′,c′] Pb[r″+i,c″+j] indicates the input    image A pixel that is offset from the corresponding pixel Pa    [r″,c″].

The size of the kernel 104 (3×3, 3×4, 4×4, 5×5, 4×7, 12×14 or the like)determines how much of a pattern is looked at when determining how wellthe overlapping source images 16 for a particular pixel match. Thelarger the kernel, the more precise the pattern match will be. However,the larger the kernel, the longer the algorithm will take to run.

As will be understood by one skilled in the art, the pixels of thegeo-referenced overlapping source images 16 may not be perfectlyaligned, but are usually within one or two pixels of alignment. Toaccount for this, the above kernel algorithm (Equation 2) is run on thedirect corresponding pixel and also on nearby surrounding pixels, forexample, within 1-3 pixels of the direct corresponding pixel. Pixelsthat represent ground locations will usually only be offset by one ortwo pixels. However, pixels that represent structures that are above theground will be offset by a significant number of pixels, or will beoccluded, and either way, will not get a good match, and therefore a badground confidence score, since they will either be two differentfeatures (when occluded) or will be two different parts of features(when they are too far away to test the same point in each).

Thus, the kernel algorithm of Equation 2 is run on the directcorresponding pixel and also on nearby surrounding pixels and the pixelscore for each run of the kernel algorithm is initially stored, and thenthe initially stored pixel scores are compared to determine the bestscore. In the embodiment of Equation 2, the best score will be thelowest score. However, Equation 2 can be modified to make the best scorethe highest score.

The direct corresponding pixel is found by calculating the geographiclocation of the pixel in the ground confidence map 100 that a groundconfidence score (using the origin and pixel size formulas from earlyon) is being generated for and then determining which pixel(s) thatlocation corresponds to in the overlapping source images 16, using theprojective equations of the source images 16. Again, because of the lackof perfectly precise data, the resulting row and column calculated maynot correspond to the actual location, which is why the surroundingpixels are checked as well. This is typically done in a 1-pixel or2-pixel radius from the corresponding pixel location. This radius needsto be large enough to account for the most common pixel location error.However, the larger the radius, the more computer time is necessary. Inaddition, if too large of a radius is used, then it will start to matchfor some things off the ground by a small amount, such as automobiles orone-story buildings.

FIG. 8 depicts a blown up portion 114 of the source images 16 within thekernel 104 a taken from vantage point 106 a of camera 105 a and vantagepoint 107 a of camera 105 b. Each pixel within the kernel 104 a has acomposite pixel score calculated in accordance with Equation 2 above. Acomparison of the pixels and composite pixel scores located within thekernel 104 a reveals that, depending on which vantage point (106 a or107 a) captured the source image 16, the pixels are associated withsubstantially different colors and thus pixel values. As previouslystated, this difference in color is indicative that, while thegeo-referenced location of the pixels are the same within the identifiedkernel 104 a, the same object is not being captured or representedwithin each source image 16. As previously shown in FIG. 6, thedifference in color is due to the camera 105 a capturing the roof of thebuilding 112 with vantage point 106 a rather than the ground asinitially projected. By comparing the composite pixel scores associatedwith each pixel located within the kernel 104 taken from vantage points106 a and 107 a, three-dimensional objects, such as the building 112 (ora tree or an automobile), can be identified. Accordingly, the groundconfidence score for the kernel 104 a indicates that the pixels of bothsource images 16 within the kernel 104 a do not represent the ground.Consequently, the pixel in the ground confidence map 100 would not be aviable candidate for designating as preferred routes 24, as thestatistical probability of cutting through a three-dimensional object,such as building 112, are elevated as indicated by the difference incolors of the pixels based on variance of the pixel scores.

FIG. 9 depicts a pixel image 116 of a plurality of pixels located withinkernel 104 b taken from vantage point 106 b of camera 105 a and vantagepoint 107 b of camera 105 b. As discussed above with reference to FIG.8, a comparison of the pixels associated with kernel 104 b utilizingEquation 2 indicate that such pixels represent the ground. This can bediscerned by similarity in pixel scores and colors of the pixels withinkernel 104 b. Consequently, the pixel in the ground confidence maprepresenting the center of the kernel 104 b is a viable candidate forbeing designated as a preferred route 24 as the statistical probabilityof cutting through a three-dimensional object, such as the building 112,is minimal.

An important aspect of the invention is the setting of a threshold valuewhich is an acceptable margin of error associated with the compositepixel scores of pixels within a particular kernel 104. While thecapturing of source images 16 is extremely precise, the capturing is notexact. It is preferable to create a threshold value for comparing thecomposite pixel scores, in which the composite pixel scores will beconsidered similar assuming the pixel does not deviate either above orbelow the pre-determined threshold value.

After the composite pixel score is determined for each pixel within theground confidence map 100, each pixel is marked in the ground confidencemap 100 by storing a pixel value indicative of the ground confidencescore calculated for the particular pixel. Once the ground confidencemap 100 has been constructed, the method of cut line steering can beaccomplished in the same manner as previously described, except thepreferred routes 24 are determined by contiguous pixels indicative ofbeing on the ground as determined by the method described above.

While this invention discusses using captured images as source images 16for input to the output mosaic image 10, it is not actually required. Itis possible to use a projected image as input to this process or even touse another output mosaic image 10 as input to this process.

The one or more assignment maps 41, ground confidence maps 100, andoutput mosaic image(s) 10 and its corresponding data are then stored onone or more computer readable medium. The one or more assignment maps41, ground confidence maps 100, and output mosaic image 10 can be storedin any format, including one of many industry standard image formatssuch as TIFF, JFIF, TARGA, Windows Bitmap File, PNG or any otherindustry standard format. The georeferencing information about theoutput mosaic image 10 might also be stored, either in a separategeoreferencing file, such as an ESRI World File, or in the same file.For example, the georeferencing information can be stored in the samefile through use of metadata tags within the file format, such as theindustry standard GeoTIFF tags used in the standard TIFF format.

It should be understood that the processes described above can beperformed with the aid of a computer system running image processingsoftware adapted to perform the functions described above, and hardwareor software embodying the logic of the processes described herein, aswell as the resulting images and data are stored on one or more computerreadable mediums. Examples of a computer readable medium include anoptical storage device, a magnetic storage device, an electronic storagedevice or the like. The term “Computer System” as used herein means asystem or systems that are able to embody and/or execute the logic ofthe processes described herein. The logic embodied in the form ofsoftware instructions or firmware for steering the cut-lines or creatingthe ground confidence map 100 may be executed on any appropriatehardware which may be a dedicated system or systems, or a generalpurpose computer system, or distributed processing computer system, allof which are well understood in the art, and a detailed description ofhow to make or use such computers is not deemed necessary herein. Whenthe computer system is used to execute the logic of the processesdescribed herein, such computer(s) and/or execution can be conducted ata same geographic location or multiple different geographic locations.Furthermore, the execution of the logic can be conducted continuously orat multiple discrete times. Further, such logic can be performed aboutsimultaneously with the capture of the images, or thereafter orcombinations thereof.

Although the foregoing invention has been described in some detail byway of illustration and example for purposes of clarity ofunderstanding, it will be obvious to those skilled in the art thatcertain changes and modifications may be practiced without departingfrom the spirit and scope thereof, as described in this specificationand as defined in the appended claims below.

What is claimed is:
 1. An apparatus for creating a mosaic image,comprising: one or more non-transitory computer readable medium storingimage processing software that when executed perform functions of:creating a mosaic image of two or more geo-referenced source images, thegeo-referenced source images having a same orientation, based on aground confidence map created by analyzing pixels of one or more of thegeo-referenced source images, the ground confidence map having valuesand data indicative of particular geographic locations represented bythe values, at least one of the values indicative of a statisticalprobability that the particular geographic locations represented by thevalues represents the ground; and using routes for steering mosaic cutlines based at least in part on the values indicative of the statisticalprobability that the particular geographic locations represented by thevalues represents the ground of the ground confidence map, such that theroutes have an increased statistical probability of cutting throughpixels representative of the ground versus routes not based on theground confidence map.
 2. The apparatus of claim 1, wherein theorientation of the geo-referenced source images is a relativeorientation of a virtual camera.
 3. The apparatus of claim 1, whereinthe orientation of the geo-referenced source images comprises an obliquedownward angle and a compass direction of an optical axis.
 4. Theapparatus of claim 1, wherein the ground confidence map is an imagefile.
 5. The apparatus of claim 1, wherein the ground confidence map hasa plurality of pixels with each pixel corresponding to a particulargeographic location.
 6. The apparatus of claim 1, wherein the one ormore geo-referenced source images include one or more oblique images. 7.The apparatus of claim 1, wherein the one or more geo-referenced sourceimages include one or more nadir images.
 8. The apparatus of claim 1,wherein the one or more geo-referenced source images include one or morecaptured image.
 9. The apparatus of claim 1, wherein the one or moregeo-referenced source images include one or more projected image. 10.The apparatus of claim 1, wherein the mosaic image is an oblique mosaicimage.
 11. A method for creating a mosaic image, comprising: creating amosaic image of two or more geo-referenced source images, thegeo-referenced source images having a same orientation, based on aground confidence map created by analyzing pixels of one or more of thegeo-referenced source images, the ground confidence map having valuesand data indicative of particular geographic locations represented bythe values, at least one of the values indicative of a statisticalprobability that the particular geographic locations represented by thevalues represents the ground; and using routes for steering mosaic cutlines based at least in part on the values indicative of the statisticalprobability that the particular geographic locations represented by thevalues represents the ground of the ground confidence map, such that theroutes have an increased statistical probability of cutting throughpixels representative of the ground versus routes not based on theground confidence map.
 12. The method of claim 11, wherein theorientation of the geo-referenced source images is a relativeorientation of a virtual camera.
 13. The method of claim 11, wherein theorientation of the geo-referenced source images comprises an obliquedownward angle and a compass direction of an optical axis.
 14. Themethod of claim 11, wherein the ground confidence map is an image file.15. The method of claim 11, wherein the ground confidence map has aplurality of pixels with each pixel corresponding to a particulargeographic location.
 16. The method of claim 11, wherein the one or moregeo-referenced source images include one or more oblique images.
 17. Themethod of claim 11, wherein the one or more geo-referenced source imagesinclude one or more nadir images.
 18. The method of claim 11, whereinthe one or more geo-referenced source images include one or morecaptured image.
 19. The method of claim 11, wherein the one or moregeo-referenced source images include one or more projected image. 20.The method of claim 11, wherein the mosaic image is an oblique mosaicimage.